/opt/course/1/contexts

CKS Simulator Kubernetes 1.28 https://killer.sh

Pre Setup

Once you've gained access to your terminal it might be wise to spend ~1 minute to setup your environment. You could set these:

alias k=kubectl # will already be pre-configured

export do="--dry-run=client -o yaml" # k create deploy nginx --image=nginx $do

export now="--force --grace-period 0" # k delete pod x $now Vim

The following settings will already be configured in your real exam environment in ~/.vimrc. But it can never hurt to be able to type these down:

set tabstop=2 set expandtab set shiftwidth=2 More setup suggestions are in the tips section.

Question 1 | Contexts

Task weight: 1%

You have access to multiple clusters from your main terminal through kubectl contexts. Write all context names into /opt/course/1/contexts, one per line.

From the kubeconfig extract the certificate of user restricted@infra-prod and write it decoded to /opt/course/1/cert.

Answer:

Maybe the fastest way is just to run:

k config get-contexts # copy by hand

k config get-contexts -o name > /opt/course/1/contexts Or using jsonpath:

k config view -o jsonpath="{.contexts[].name}" k config view -o jsonpath="{.contexts[].name}" | tr " " "\n" # new lines k config view -o jsonpath="{.contexts[*].name}" | tr " " "\n" > /opt/course/1/contexts The content could then look like:

/opt/course/1/contexts

gianna@infra-prod infra-prod restricted@infra-prod workload-prod workload-stage For the certificate we could just run

k config view --raw And copy it manually. Or we do:

k config view --raw -ojsonpath="{.users[2].user.client-certificate-data}" | base64 -d > /opt/course/1/cert Or even:

k config view --raw -ojsonpath="{.users[?(.name == 'restricted@infra-prod')].user.client-certificate-data}" | base64 -d > /opt/course/1/cert

/opt/course/1/cert

Question 2 | Runtime Security with Falco

Task weight: 4%

Use context: kubectl config use-context workload-prod

Falco is installed with default configuration on node cluster1-node1. Connect using ssh cluster1-node1. Use it to:

Find a Pod running image nginx which creates unwanted package management processes inside its container. Find a Pod running image httpd which modifies /etc/passwd. Save the Falco logs for case 1 under /opt/course/2/falco.log in format:

time-with-nanosconds,container-id,container-name,user-name No other information should be in any line. Collect the logs for at least 30 seconds.

Afterwards remove the threads (both 1 and 2) by scaling the replicas of the Deployments that control the offending Pods down to 0.

Answer:

Falco, the open-source cloud-native runtime security project, is the de facto Kubernetes threat detection engine. NOTE: Other tools you might have to be familar with are sysdig or tracee

Use Falco as service

First we can investigate Falco config a little:

➜ ssh cluster1-node1

➜ root@cluster1-node1:~# service falco status ● falco.service - LSB: Falco syscall activity monitoring agent Loaded: loaded (/etc/init.d/falco; generated) Active: active (running) since Sat 2020-10-10 06:36:15 UTC; 2h 1min ago ...

➜ root@cluster1-node1:~# cd /etc/falco

➜ root@cluster1-node1:/etc/falco# ls falco.yaml falco_rules.local.yaml falco_rules.yaml k8s_audit_rules.yaml rules.available rules.d This is the default configuration, if we look into falco.yaml we can see:

/etc/falco/falco.yaml

...

Where security notifications should go.

Multiple outputs can be enabled.

syslog_output: enabled: true ... This means that Falco is writing into syslog, hence we can do:

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco Sep 15 08:44:04 ubuntu2004 falco: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d) Sep 15 08:44:04 ubuntu2004 falco: Falco initialized with configuration file /etc/falco/falco.yaml Sep 15 08:44:04 ubuntu2004 falco: Loading rules from file /etc/falco/falco_rules.yaml: ... Yep, quite some action going on in there. Let's investigate the first offending Pod:

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep nginx | grep process Sep 16 06:23:47 ubuntu2004 falco: 06:23:47.376241377: Error Package management process launched in container (user=root user_loginuid=-1 command=apk container_id=7a5ea6a080d1 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine) ...

➜ root@cluster1-node1:~# crictl ps -id 7a5ea6a080d1 CONTAINER ID IMAGE NAME ... POD ID 7a5ea6a080d1b 6f715d38cfe0e nginx ... 7a864406b9794

root@cluster1-node1:~# crictl pods -id 7a864406b9794 POD ID ... NAME NAMESPACE ... 7a864406b9794 ... webapi-6cfddcd6f4-ftxg4 team-blue ... First Pod is webapi-6cfddcd6f4-ftxg4 in Namespace team-blue.

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep httpd | grep passwd Sep 16 06:23:48 ubuntu2004 falco: 06:23:48.830962378: Error File below /etc opened for writing (user=root user_loginuid=-1 command=sed -i $d /etc/passwd parent=sh pcmdline=sh -c echo hacker >> /etc/passwd; sed -i '$d' /etc/passwd; true file=/etc/passwdngFmAl program=sed gparent= ggparent= gggparent= container_id=b1339d5cc2de image=docker.io/library/httpd)

➜ root@cluster1-node1:~# crictl ps -id b1339d5cc2de CONTAINER ID IMAGE NAME ... POD ID b1339d5cc2dee f6b40f9f8ad71 httpd ... 595af943c3245

root@cluster1-node1:~# crictl pods -id 595af943c3245 POD ID ... NAME NAMESPACE ... 595af943c3245 ... rating-service-68cbdf7b7-v2p6g team-purple ... Second Pod is rating-service-68cbdf7b7-v2p6g in Namespace team-purple.

Eliminate offending Pods

The logs from before should allow us to find and "eliminate" the offending Pods:

➜ k get pod -A | grep webapi team-blue webapi-6cfddcd6f4-ftxg4 1/1 Running

➜ k -n team-blue scale deploy webapi --replicas 0 deployment.apps/webapi scaled

➜ k get pod -A | grep rating-service team-purple rating-service-68cbdf7b7-v2p6g 1/1 Running

➜ k -n team-purple scale deploy rating-service --replicas 0 deployment.apps/rating-service scaled

Use Falco from command line

We can also use Falco directly from command line, but only if the service is disabled:

➜ root@cluster1-node1:~# service falco stop

➜ root@cluster1-node1:~# falco Thu Sep 16 06:33:11 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d) Thu Sep 16 06:33:11 2021: Falco initialized with configuration file /etc/falco/falco.yaml Thu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/falco_rules.yaml: Thu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/falco_rules.local.yaml: Thu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/k8s_audit_rules.yaml: Thu Sep 16 06:33:12 2021: Starting internal webserver, listening on port 8765 06:33:17.382603204: Error Package management process launched in container (user=root user_loginuid=-1 command=apk container_id=7a5ea6a080d1 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine) ... We can see that rule files are loaded and logs printed afterwards.

Create logs in correct format

The task requires us to store logs for "unwanted package management processes" in format time,container-id,container-name,user-name. The output from falco shows entries for "Error Package management process launched" in a default format. Let's find the proper file that contains the rule and change it:

➜ root@cluster1-node1:~# cd /etc/falco/

➜ root@cluster1-node1:/etc/falco# grep -r "Package management process launched" . ./falco_rules.yaml: Package management process launched in container (user=%user.name user_loginuid=%user.loginuid

➜ root@cluster1-node1:/etc/falco# cp falco_rules.yaml falco_rules.yaml_ori

➜ root@cluster1-node1:/etc/falco# vim falco_rules.yaml Find the rule which looks like this:

Container is supposed to be immutable. Package management should be done in building the image.

rule: Launch Package Management Process in Container desc: Package management process ran inside container condition: > spawned_process and container and user.name != "_apt" and package_mgmt_procs and not package_mgmt_ancestor_procs and not user_known_package_manager_in_container output: > Package management process launched in container (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag) priority: ERROR tags: [process, mitre_persistence] Should be changed into the required format:

Container is supposed to be immutable. Package management should be done in building the image.

rule: Launch Package Management Process in Container desc: Package management process ran inside container condition: > spawned_process and container and user.name != "_apt" and package_mgmt_procs and not package_mgmt_ancestor_procs and not user_known_package_manager_in_container output: > Package management process launched in container %evt.time,%container.id,%container.name,%user.name priority: ERROR tags: [process, mitre_persistence] For all available fields we can check https://falco.org/docs/rules/supported-fields, which should be allowed to open during the exam.

Next we check the logs in our adjusted format:

➜ root@cluster1-node1:/etc/falco# falco | grep "Package management"

06:38:28.077150666: Error Package management process launched in container 06:38:28.077150666,090aad374a0a,nginx,root 06:38:33.058263010: Error Package management process launched in container 06:38:33.058263010,090aad374a0a,nginx,root 06:38:38.068693625: Error Package management process launched in container 06:38:38.068693625,090aad374a0a,nginx,root 06:38:43.066159360: Error Package management process launched in container 06:38:43.066159360,090aad374a0a,nginx,root 06:38:48.059792139: Error Package management process launched in container 06:38:48.059792139,090aad374a0a,nginx,root 06:38:53.063328933: Error Package management process launched in container 06:38:53.063328933,090aad374a0a,nginx,root This looks much better. Copy&paste the output into file /opt/course/2/falco.log on your main terminal. The content should be cleaned like this:

/opt/course/2/falco.log

06:38:28.077150666,090aad374a0a,nginx,root 06:38:33.058263010,090aad374a0a,nginx,root 06:38:38.068693625,090aad374a0a,nginx,root 06:38:43.066159360,090aad374a0a,nginx,root 06:38:48.059792139,090aad374a0a,nginx,root 06:38:53.063328933,090aad374a0a,nginx,root 06:38:58.070912841,090aad374a0a,nginx,root 06:39:03.069592140,090aad374a0a,nginx,root 06:39:08.064805371,090aad374a0a,nginx,root 06:39:13.078109098,090aad374a0a,nginx,root 06:39:18.065077287,090aad374a0a,nginx,root 06:39:23.061012151,090aad374a0a,nginx,root For a few entries it should be fast to just clean it up manually. If there are larger amounts of entries we could do:

cat /opt/course/2/falco.log.dirty | cut -d" " -f 9 > /opt/course/2/falco.log The tool cut will split input into fields using space as the delimiter (-d""). We then only select the 9th field using -f 9.

Local falco rules

There is also a file /etc/falco/falco_rules.local.yaml in which we can override existing default rules. This is a much cleaner solution for production. Choose the faster way for you in the exam if nothing is specified in the task.

Question 3 | Apiserver Security

Task weight: 3%

Use context: kubectl config use-context workload-prod

You received a list from the DevSecOps team which performed a security investigation of the k8s cluster1 (workload-prod). The list states the following about the apiserver setup:

Accessible through a NodePort Service Change the apiserver setup so that:

Only accessible through a ClusterIP Service

Answer:

In order to modify the parameters for the apiserver, we first ssh into the master node and check which parameters the apiserver process is running with:

➜ ssh cluster1-controlplane1

➜ root@cluster1-controlplane1:~# ps aux | grep kube-apiserver root 27622 7.4 15.3 1105924 311788 ? Ssl 10:31 11:03 kube-apiserver --advertise-address=192.168.100.11 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubernetes-service-node-port=31000 --proxy-client-cert- ... We may notice the following argument:

--kubernetes-service-node-port=31000 We can also check the Service and see it's of type NodePort:

➜ root@cluster1-controlplane1:~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes NodePort 10.96.0.1 443:31000/TCP 5d2h The apiserver runs as a static Pod, so we can edit the manifest. But before we do this we also create a copy in case we mess things up:

➜ root@cluster1-controlplane1:~# cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/3_kube-apiserver.yaml

➜ root@cluster1-controlplane1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml We should remove the unsecure settings:

/etc/kubernetes/manifests/kube-apiserver.yaml

apiVersion: v1 kind: Pod metadata: annotations: kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.100.11:6443 creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers:

command:
- kube-apiserver
- --advertise-address=192.168.100.11
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

- --kubernetes-service-node-port=31000 # delete or set to 0

- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key

... Once the changes are made, give the apiserver some time to start up again. Check the apiserver's Pod status and the process parameters:

➜ root@cluster1-controlplane1:~# kubectl -n kube-system get pod | grep apiserver kube-apiserver-cluster1-controlplane1 1/1 Running 0 38s

➜ root@cluster1-controlplane1:~# ps aux | grep kube-apiserver | grep node-port The apiserver got restarted without the unsecure settings. However, the Service kubernetes will still be of type NodePort:

➜ root@cluster1-controlplane1:~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes NodePort 10.96.0.1 443:31000/TCP 5d3h We need to delete the Service for the changes to take effect:

➜ root@cluster1-controlplane1:~# kubectl delete svc kubernetes service "kubernetes" deleted After a few seconds:

➜ root@cluster1-controlplane1:~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 6s This should satisfy the DevSecOps team.

Question 4 | Pod Security Standard

Task weight: 8%

Use context: kubectl config use-context workload-prod

There is Deployment container-host-hacker in Namespace team-red which mounts /run/containerd as a hostPath volume on the Node where it's running. This means that the Pod can access various data about other containers running on the same Node.

To prevent this configure Namespace team-red to enforce the baseline Pod Security Standard. Once completed, delete the Pod of the Deployment mentioned above.

Check the ReplicaSet events and write the event/log lines containing the reason why the Pod isn't recreated into /opt/course/4/logs.

Answer:

Making Namespaces use Pod Security Standards works via labels. We can simply edit it:

k edit ns team-red Now we configure the requested label:

kubectl edit namespace team-red

apiVersion: v1 kind: Namespace metadata: labels: kubernetes.io/metadata.name: team-red pod-security.kubernetes.io/enforce: baseline # add name: team-red ... This should already be enough for the default Pod Security Admission Controller to pick up on that change. Let's test it and delete the Pod to see if it'll be recreated or fails, it should fail!

➜ k -n team-red get pod NAME READY STATUS RESTARTS AGE container-host-hacker-dbf989777-wm8fc 1/1 Running 0 115s

➜ k -n team-red delete pod container-host-hacker-dbf989777-wm8fc pod "container-host-hacker-dbf989777-wm8fc" deleted

➜ k -n team-red get pod No resources found in team-red namespace. Usually the ReplicaSet of a Deployment would recreate the Pod if deleted, here we see this doesn't happen. Let's check why:

➜ k -n team-red get rs NAME DESIRED CURRENT READY AGE container-host-hacker-dbf989777 1 0 0 5m25s

➜ k -n team-red describe rs container-host-hacker-dbf989777 Name: container-host-hacker-dbf989777 Namespace: team-red ... Events: Type Reason Age From Message

... Warning FailedCreate 2m41s replicaset-controller Error creating: pods "container-host-hacker-dbf989777-bjwgv" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "containerdata") Warning FailedCreate 2m2s (x9 over 2m40s) replicaset-controller (combined from similar events): Error creating: pods "container-host-hacker-dbf989777-kjfpn" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "containerdata") There we go! Finally we write the reason into the requested file so that Mr Scoring will be happy too!

/opt/course/4/logs

Warning FailedCreate 2m2s (x9 over 2m40s) replicaset-controller (combined from similar events): Error creating: pods "container-host-hacker-dbf989777-kjfpn" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "containerdata") Pod Security Standards can give a great base level of security! But when one finds themselves wanting to deeper adjust the levels like baseline or restricted... this isn't possible and 3rd party solutions like OPA could be looked at.

Question 5 | CIS Benchmark

Task weight: 3%

Use context: kubectl config use-context infra-prod

You're ask to evaluate specific settings of cluster2 against the CIS Benchmark recommendations. Use the tool kube-bench which is already installed on the nodes.

Connect using ssh cluster2-controlplane1 and ssh cluster2-node1.

On the master node ensure (correct if necessary) that the CIS recommendations are set for:

The --profiling argument of the kube-controller-manager The ownership of directory /var/lib/etcd On the worker node ensure (correct if necessary) that the CIS recommendations are set for:

The permissions of the kubelet configuration /var/lib/kubelet/config.yaml The --client-ca-file argument of the kubelet

Answer:

Number 1

First we ssh into the master node run kube-bench against the master components:

➜ ssh cluster2-controlplane1

➜ root@cluster2-controlplane1:~# kube-bench run --targets=master ... == Summary == 41 checks PASS 13 checks FAIL 11 checks WARN 0 checks INFO We see some passes, fails and warnings. Let's check the required task (1) of the controller manager:

➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep kube-controller -A 3 1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold, for example: --terminated-pod-gc-threshold=10

1.3.2 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the master node and set the below parameter. --profiling=false

1.3.6 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. --feature-gates=RotateKubeletServerCertificate=true There we see 1.3.2 which suggests to set --profiling=false, so we obey:

➜ root@cluster2-controlplane1:~# vim /etc/kubernetes/manifests/kube-controller-manager.yaml Edit the corresponding line:

/etc/kubernetes/manifests/kube-controller-manager.yaml

apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-controller-manager tier: control-plane name: kube-controller-manager namespace: kube-system spec: containers:

command:
- kube-controller-manager
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
- --bind-address=127.0.0.1
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --cluster-cidr=10.244.0.0/16
- --cluster-name=kubernetes
- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
- --controllers=*,bootstrapsigner,tokencleaner
- --kubeconfig=/etc/kubernetes/controller-manager.conf
- --leader-elect=true
- --node-cidr-mask-size=24
- --port=0
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --root-ca-file=/etc/kubernetes/pki/ca.crt
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/12
- --use-service-account-credentials=true
- --profiling=false # add ... We wait for the Pod to restart, then run kube-bench again to check if the problem was solved:

➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep kube-controller -A 3 1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold, for example: --terminated-pod-gc-threshold=10

1.3.6 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. --feature-gates=RotateKubeletServerCertificate=true Problem solved and 1.3.2 is passing:

root@cluster2-controlplane1:~# kube-bench run --targets=master | grep 1.3.2 [PASS] 1.3.2 Ensure that the --profiling argument is set to false (Scored)

Number 2

Next task (2) is to check the ownership of directory /var/lib/etcd, so we first have a look:

➜ root@cluster2-controlplane1:~# ls -lh /var/lib | grep etcd drwx------ 3 root root 4.0K Sep 11 20:08 etcd Looks like user root and group root. Also possible to check using:

➜ root@cluster2-controlplane1:~# stat -c %U:%G /var/lib/etcd root:root But what has kube-bench to say about this?

➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep "/var/lib/etcd" -B5

1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the below command: ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, chown etcd:etcd /var/lib/etcd To comply we run the following:

➜ root@cluster2-controlplane1:~# chown etcd:etcd /var/lib/etcd

➜ root@cluster2-controlplane1:~# ls -lh /var/lib | grep etcd drwx------ 3 etcd etcd 4.0K Sep 11 20:08 etcd This looks better. We run kube-bench again, and make sure test 1.1.12. is passing.

➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep 1.1.12 [PASS] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Scored) Done.

Number 3

To continue with number (3), we'll head to the worker node and ensure that the kubelet configuration file has the minimum necessary permissions as recommended:

➜ ssh cluster2-node1

➜ root@cluster2-node1:~# kube-bench run --targets=node ... == Summary == 13 checks PASS 10 checks FAIL 2 checks WARN 0 checks INFO Also here some passes, fails and warnings. We check the permission level of the kubelet config file:

➜ root@cluster2-node1:~# stat -c %a /var/lib/kubelet/config.yaml 777 777 is highly permissive access level and not recommended by the kube-bench guidelines:

➜ root@cluster2-node1:~# kube-bench run --targets=node | grep /var/lib/kubelet/config.yaml -B2

4.1.9 Run the following command (using the config file location identified in the Audit step) chmod 644 /var/lib/kubelet/config.yaml We obey and set the recommended permissions:

➜ root@cluster2-node1:~# chmod 644 /var/lib/kubelet/config.yaml

➜ root@cluster2-node1:~# stat -c %a /var/lib/kubelet/config.yaml 644 And check if test 2.2.10 is passing:

➜ root@cluster2-node1:~# kube-bench run --targets=node | grep 4.1.9 [PASS] 2.2.10 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)

Number 4

Finally for number (4), let's check whether --client-ca-file argument for the kubelet is set properly according to kube-bench recommendations:

➜ root@cluster2-node1:~# kube-bench run --targets=node | grep client-ca-file [PASS] 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) This looks passing with 4.2.3. The other ones are about the file that the parameter points to and can be ignored here.

To further investigate we run the following command to locate the kubelet config file, and open it:

➜ root@cluster2-node1:~# ps -ef | grep kubelet root 5157 1 2 20:28 ? 00:03:22 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2 root 19940 11901 0 22:38 pts/0 00:00:00 grep --color=auto kubelet

➜ root@croot@cluster2-node1:~# vim /var/lib/kubelet/config.yaml

/var/lib/kubelet/config.yaml

apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt ... The clientCAFile points to the location of the certificate, which is correct.

Question 6 | Verify Platform Binaries

Task weight: 2%

(can be solved in any kubectl context)

There are four Kubernetes server binaries located at /opt/course/6/binaries. You're provided with the following verified sha512 values for these:

kube-apiserver f417c0555bc0167355589dd1afe23be9bf909bf98312b1025f12015d1b58a1c62c9908c0067a7764fa35efdac7016a9efa8711a44425dd6692906a7c283f032c

kube-controller-manager 60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60

kube-proxy 52f9d8ad045f8eee1d689619ef8ceef2d86d50c75a6a332653240d7ba5b2a114aca056d9e513984ade24358c9662714973c1960c62a5cb37dd375631c8a614c6

kubelet 4be40f2440619e990897cf956c32800dc96c2c983bf64519854a3309fa5aa21827991559f9c44595098e27e6f2ee4d64a3fdec6baba8a177881f20e3ec61e26c

Delete those binaries that don't match with the sha512 values above.

Answer:

We check the directory:

➜ cd /opt/course/6/binaries

➜ ls kube-apiserver kube-controller-manager kube-proxy kubelet To generate the sha512 sum of a binary we do:

➜ sha512sum kube-apiserver f417c0555bc0167355589dd1afe23be9bf909bf98312b1025f12015d1b58a1c62c9908c0067a7764fa35efdac7016a9efa8711a44425dd6692906a7c283f032c kube-apiserver Looking good, next:

➜ sha512sum kube-controller-manager 60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33b0a8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60 kube-controller-manager Okay, next:

➜ sha512sum kube-proxy 52f9d8ad045f8eee1d689619ef8ceef2d86d50c75a6a332653240d7ba5b2a114aca056d9e513984ade24358c9662714973c1960c62a5cb37dd375631c8a614c6 kube-proxy Also good, and finally:

➜ sha512sum kubelet 7b720598e6a3483b45c537b57d759e3e82bc5c53b3274f681792f62e941019cde3d51a7f9b55158abf3810d506146bc0aa7cf97b36f27f341028a54431b335be kubelet Catch! Binary kubelet has a different hash!

But did we actually compare everything properly before? Let's have a closer look at kube-controller-manager again:

➜ sha512sum kube-controller-manager > compare

➜ vim compare Edit to only have the provided hash and the generated one in one line each:

./compare

60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33b0a8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60
60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60 Looks right at a first glance, but if we do:

➜ cat compare | uniq 60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33b0a8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60 60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60 This shows they are different, by just one character actually.

To complete the task we do:

rm kubelet kube-controller-manager

Question 7 | Open Policy Agent

Task weight: 6%

Use context: kubectl config use-context infra-prod

The Open Policy Agent and Gatekeeper have been installed to, among other things, enforce blacklisting of certain image registries. Alter the existing constraint and/or template to also blacklist images from very-bad-registry.com.

Test it by creating a single Pod using image very-bad-registry.com/image in Namespace default, it shouldn't work.

You can also verify your changes by looking at the existing Deployment untrusted in Namespace default, it uses an image from the new untrusted source. The OPA contraint should throw violation messages for this one.

Answer:

We look at existing OPA constraints, these are implemeted using CRDs by Gatekeeper:

➜ k get crd NAME CREATED AT blacklistimages.constraints.gatekeeper.sh 2020-09-14T19:29:31Z configs.config.gatekeeper.sh 2020-09-14T19:29:04Z constraintpodstatuses.status.gatekeeper.sh 2020-09-14T19:29:05Z constrainttemplatepodstatuses.status.gatekeeper.sh 2020-09-14T19:29:05Z constrainttemplates.templates.gatekeeper.sh 2020-09-14T19:29:05Z requiredlabels.constraints.gatekeeper.sh 2020-09-14T19:29:31Z So we can do:

➜ k get constraint NAME AGE blacklistimages.constraints.gatekeeper.sh/pod-trusted-images 10m

NAME AGE requiredlabels.constraints.gatekeeper.sh/namespace-mandatory-labels 10m and then look at the one that is probably about blacklisting images:

k edit blacklistimages pod-trusted-images

kubectl edit blacklistimages pod-trusted-images

apiVersion: constraints.gatekeeper.sh/v1beta1 kind: BlacklistImages metadata: ... spec: match: kinds: - apiGroups: - "" kinds: - Pod It looks like this constraint simply applies the template to all Pods, no arguments passed. So we edit the template:

k edit constrainttemplates blacklistimages

kubectl edit constrainttemplates blacklistimages

apiVersion: templates.gatekeeper.sh/v1beta1 kind: ConstraintTemplate metadata: ... spec: crd: spec: names: kind: BlacklistImages targets:

rego: | package k8strustedimages

images { image := input.review.object.spec.containers[_].image not startswith(image, "docker-fake.io/") not startswith(image, "google-gcr-fake.com/") not startswith(image, "very-bad-registry.com/") # ADD THIS LINE }

violation[{"msg": msg}] { not images msg := "not trusted image!" } target: admission.k8s.gatekeeper.sh We simply have to add another line. After editing we try to create a Pod of the bad image:

➜ k run opa-test --image=very-bad-registry.com/image Error from server ([denied by pod-trusted-images] not trusted image!): admission webhook "validation.gatekeeper.sh" denied the request: [denied by pod-trusted-images] not trusted image! Nice! After some time we can also see that Pods of the existing Deployment "untrusted" will be listed as violators:

➜ k describe blacklistimages pod-trusted-images ... Total Violations: 2 Violations: Enforcement Action: deny Kind: Namespace Message: you must provide labels: {"security-level"} Name: sidecar-injector Enforcement Action: deny Kind: Pod Message: not trusted image! Name: untrusted-68c4944d48-tfsnb Namespace: default Events: Great, OPA fights bad registries !

Question 8 | Secure Kubernetes Dashboard

Task weight: 3%

Use context: kubectl config use-context workload-prod

The Kubernetes Dashboard is installed in Namespace kubernetes-dashboard and is configured to:

Allow users to "skip login" Allow insecure access (HTTP without authentication) Allow basic authentication Allow access from outside the cluster You are asked to make it more secure by:

Deny users to "skip login" Deny insecure access, enforce HTTPS (self signed certificates are ok for now) Add the --auto-generate-certificates argument Enforce authentication using a token (with possibility to use RBAC) Allow only cluster internal access

Answer:

Head to https://github.com/kubernetes/dashboard/tree/master/docs to find documentation about the dashboard. This link is not on the allowed list of urls during the real exam. This means you should be provided will all information necessary in case of a task like this.

First we have a look in Namespace kubernetes-dashboard:

➜ k -n kubernetes-dashboard get pod,svc NAME READY STATUS RESTARTS AGE pod/dashboard-metrics-scraper-7b59f7d4df-fbpd9 1/1 Running 0 24m pod/kubernetes-dashboard-6d8cd5dd84-w7wr2 1/1 Running 0 24m

NAME TYPE ... PORT(S) AGE service/dashboard-metrics-scraper ClusterIP ... 8000/TCP 24m service/kubernetes-dashboard NodePort ... 9090:32520/TCP,443:31206/TCP 24m We can see one running Pod and a NodePort Service exposing it. Let's try to connect to it via a NodePort, we can use IP of any Node:

(your port might be a different)

➜ k get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP ... cluster1-controlplane1 Ready master 37m v1.28.2 192.168.100.11 ... cluster1-node1 Ready 36m v1.28.2 192.168.100.12 ... cluster1-node2 Ready 34m v1.28.2 192.168.100.13 ...

➜ curl http://192.168.100.11:32520

ceb8989101b

Successfully tagged registry.killer.sh:5000/image-verify:v2 ceb8989101bccd9f6b9c3b4c6c75f6c3561f19a5b784edd1f1a36fa0fb34a9df We can then test our changes by running the container locally:

➜ :/opt/course/16/image$ podman run registry.killer.sh:5000/image-verify:v2 Thu Sep 16 06:01:47 UTC 2021 uid=101(myuser) gid=102(myuser) groups=102(myuser)

Thu Sep 16 06:01:48 UTC 2021 uid=101(myuser) gid=102(myuser) groups=102(myuser)

Thu Sep 16 06:01:49 UTC 2021 uid=101(myuser) gid=102(myuser) groups=102(myuser) Looking good, so we push:

➜ :/opt/course/16/image$ podman push registry.killer.sh:5000/image-verify:v2 Getting image source signatures Copying blob cd0853834d88 done
Copying blob 5298d0709c3e skipped: already exists
Copying blob e6688e911f15 done
Copying blob dbc406096645 skipped: already exists
Copying blob 98895ed393d9 done
Copying config ceb8989101 done
Writing manifest to image destination Storing signatures And we update the Deployment to use the new image:

k -n team-blue edit deploy image-verify

kubectl -n team-blue edit deploy image-verify

apiVersion: apps/v1 kind: Deployment metadata: ... spec: ... template: ... spec: containers: - image: registry.killer.sh:5000/image-verify:v2 # change And afterwards we can verify our changes by looking at the Pod logs:

➜ k -n team-blue logs -f -l id=image-verify Fri Sep 25 21:06:55 UTC 2020 uid=101(myuser) gid=102(myuser) groups=102(myuser) Also to verify our changes even further:

➜ k -n team-blue exec image-verify-55fbcd4c9b-x2flc -- curl OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: "curl": executable file not found in $PATH": unknown command terminated with exit code 126

➜ k -n team-blue exec image-verify-55fbcd4c9b-x2flc -- nginx -v nginx version: nginx/1.18.0 Another task solved.

Question 17 | Audit Log Policy

Task weight: 7%

Use context: kubectl config use-context infra-prod

Audit Logging has been enabled in the cluster with an Audit Policy located at /etc/kubernetes/audit/policy.yaml on cluster2-controlplane1.

Change the configuration so that only one backup of the logs is stored.

Alter the Policy in a way that it only stores logs:

From Secret resources, level Metadata From "system:nodes" userGroups, level RequestResponse After you altered the Policy make sure to empty the log file so it only contains entries according to your changes, like using truncate -s 0 /etc/kubernetes/audit/logs/audit.log.

NOTE: You can use jq to render json more readable. cat data.json | jq

Answer:

First we check the apiserver configuration and change as requested:

➜ ssh cluster2-controlplane1

➜ root@cluster2-controlplane1:~# cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/17_kube-apiserver.yaml # backup

➜ root@cluster2-controlplane1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml

/etc/kubernetes/manifests/kube-apiserver.yaml

apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers:

command:
- kube-apiserver
- --audit-policy-file=/etc/kubernetes/audit/policy.yaml
- --audit-log-path=/etc/kubernetes/audit/logs/audit.log
- --audit-log-maxsize=5
- --audit-log-maxbackup=1 # CHANGE
- --advertise-address=192.168.100.21
- --allow-privileged=true ...

NOTE: You should know how to enable Audit Logging completely yourself as described in the docs. Feel free to try this in another cluster in this environment.

Now we look at the existing Policy:

➜ root@cluster2-controlplane1:~# vim /etc/kubernetes/audit/policy.yaml

/etc/kubernetes/audit/policy.yaml

apiVersion: audit.k8s.io/v1 kind: Policy rules:

level: Metadata We can see that this simple Policy logs everything on Metadata level. So we change it to the requirements:

/etc/kubernetes/audit/policy.yaml

apiVersion: audit.k8s.io/v1 kind: Policy rules:

log Secret resources audits, level Metadata

level: Metadata resources:
- group: "" resources: ["secrets"]

log node related audits, level RequestResponse

level: RequestResponse userGroups: ["system:nodes"]

for everything else don't log anything

level: None After saving the changes we have to restart the apiserver:

➜ root@cluster2-controlplane1:~# cd /etc/kubernetes/manifests/

➜ root@cluster2-controlplane1:/etc/kubernetes/manifests# mv kube-apiserver.yaml ..

➜ root@cluster2-controlplane1:/etc/kubernetes/manifests# watch crictl ps # wait for apiserver gone

➜ root@cluster2-controlplane1:/etc/kubernetes/manifests# truncate -s 0 /etc/kubernetes/audit/logs/audit.log

➜ root@cluster2-controlplane1:/etc/kubernetes/manifests# mv ../kube-apiserver.yaml . Once the apiserver is running again we can check the new logs and scroll through some entries:

cat audit.log | tail | jq

{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "auditID": "e598dc9e-fc8b-4213-aee3-0719499ab1bd", "stage": "RequestReceived", "requestURI": "...", "verb": "watch", "user": { "username": "system:serviceaccount:gatekeeper-system:gatekeeper-admin", "uid": "79870838-75a8-479b-ad42-4b7b75bd17a3", "groups": [ "system:serviceaccounts", "system:serviceaccounts:gatekeeper-system", "system:authenticated" ] }, "sourceIPs": [ "192.168.102.21" ], "userAgent": "manager/v0.0.0 (linux/amd64) kubernetes/$Format", "objectRef": { "resource": "secrets", "apiVersion": "v1" }, "requestReceivedTimestamp": "2020-09-27T20:01:36.238911Z", "stageTimestamp": "2020-09-27T20:01:36.238911Z", "annotations": { "authentication.k8s.io/legacy-token": "..." } } Above we logged a watch action by OPA Gatekeeper for Secrets, level Metadata.

{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "RequestResponse", "auditID": "c90e53ed-b0cf-4cc4-889a-f1204dd39267", "stage": "ResponseComplete", "requestURI": "...", "verb": "list", "user": { "username": "system:node:cluster2-controlplane1", "groups": [ "system:nodes", "system:authenticated" ] }, "sourceIPs": [ "192.168.100.21" ], "userAgent": "kubelet/v1.19.1 (linux/amd64) kubernetes/206bcad", "objectRef": { "resource": "configmaps", "namespace": "kube-system", "name": "kube-proxy", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 200 }, "responseObject": { "kind": "ConfigMapList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/namespaces/kube-system/configmaps", "resourceVersion": "83409" }, "items": [ { "metadata": { "name": "kube-proxy", "namespace": "kube-system", "selfLink": "/api/v1/namespaces/kube-system/configmaps/kube-proxy", "uid": "0f1c3950-430a-4543-83e4-3f9c87a478b8", "resourceVersion": "232", "creationTimestamp": "2020-09-26T20:59:50Z", "labels": { "app": "kube-proxy" }, "annotations": { "kubeadm.kubernetes.io/component-config.hash": "..." }, "managedFields": [ { ... } ] }, ... } ] }, "requestReceivedTimestamp": "2020-09-27T20:01:36.223781Z", "stageTimestamp": "2020-09-27T20:01:36.225470Z", "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "" } } And in the one above we logged a list action by system:nodes for a ConfigMaps, level RequestResponse.

Because all JSON entries are written in a single line in the file we could also run some simple verifications on our Policy:

shows Secret entries

cat audit.log | grep '"resource":"secrets"' | wc -l

confirms Secret entries are only of level Metadata

cat audit.log | grep '"resource":"secrets"' | grep -v '"level":"Metadata"' | wc -l

shows RequestResponse level entries

cat audit.log | grep -v '"level":"RequestResponse"' | wc -l

shows RequestResponse level entries are only for system:nodes

cat audit.log | grep '"level":"RequestResponse"' | grep -v "system:nodes" | wc -l Looks like our job is done.

Question 18 | Investigate Break-in via Audit Log

Task weight: 4%

Use context: kubectl config use-context infra-prod

Namespace security contains five Secrets of type Opaque which can be considered highly confidential. The latest Incident-Prevention-Investigation revealed that ServiceAccount p.auster had too broad access to the cluster for some time. This SA should've never had access to any Secrets in that Namespace.

Find out which Secrets in Namespace security this SA did access by looking at the Audit Logs under /opt/course/18/audit.log.

Change the password to any new string of only those Secrets that were accessed by this SA.

NOTE: You can use jq to render json more readable. cat data.json | jq

Answer:

First we look at the Secrets this is about:

➜ k -n security get secret | grep Opaque kubeadmin-token Opaque 1 37m mysql-admin Opaque 1 37m postgres001 Opaque 1 37m postgres002 Opaque 1 37m vault-token Opaque 1 37m Next we investigate the Audit Log file:

➜ cd /opt/course/18

➜ :/opt/course/18$ ls -lh total 7.1M -rw-r--r-- 1 k8s k8s 7.5M Sep 24 21:31 audit.log

➜ :/opt/course/18$ cat audit.log | wc -l 4451 Audit Logs can be huge and it's common to limit the amount by creating an Audit Policy and to transfer the data in systems like Elasticsearch. In this case we have a simple JSON export, but it already contains 4451 lines.

We should try to filter the file down to relevant information:

➜ :/opt/course/18$ cat audit.log | grep "p.auster" | wc -l 28 Not too bad, only 28 logs for ServiceAccount p.auster.

➜ :/opt/course/18$ cat audit.log | grep "p.auster" | grep Secret | wc -l 2 And only 2 logs related to Secrets...

➜ :/opt/course/18$ cat audit.log | grep "p.auster" | grep Secret | grep list | wc -l 0

➜ :/opt/course/18$ cat audit.log | grep "p.auster" | grep Secret | grep get | wc -l 2 No list actions, which is good, but 2 get actions, so we check these out:

cat audit.log | grep "p.auster" | grep Secret | grep get | jq

{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "RequestResponse", "auditID": "74fd9e03-abea-4df1-b3d0-9cfeff9ad97a", "stage": "ResponseComplete", "requestURI": "/api/v1/namespaces/security/secrets/vault-token", "verb": "get", "user": { "username": "system:serviceaccount:security:p.auster", "uid": "29ecb107-c0e8-4f2d-816a-b16f4391999c", "groups": [ "system:serviceaccounts", "system:serviceaccounts:security", "system:authenticated" ] }, ... "userAgent": "curl/7.64.0", "objectRef": { "resource": "secrets", "namespace": "security", "name": "vault-token", "apiVersion": "v1" }, ... } { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "RequestResponse", "auditID": "aed6caf9-5af0-4872-8f09-ad55974bb5e0", "stage": "ResponseComplete", "requestURI": "/api/v1/namespaces/security/secrets/mysql-admin", "verb": "get", "user": { "username": "system:serviceaccount:security:p.auster", "uid": "29ecb107-c0e8-4f2d-816a-b16f4391999c", "groups": [ "system:serviceaccounts", "system:serviceaccounts:security", "system:authenticated" ] }, ... "userAgent": "curl/7.64.0", "objectRef": { "resource": "secrets", "namespace": "security", "name": "mysql-admin", "apiVersion": "v1" }, ... } There we see that Secrets vault-token and mysql-admin were accessed by p.auster. Hence we change the passwords for those.

➜ echo new-vault-pass | base64 bmV3LXZhdWx0LXBhc3MK

➜ k -n security edit secret vault-token

➜ echo new-mysql-pass | base64 bmV3LW15c3FsLXBhc3MK

➜ k -n security edit secret mysql-admin Audit Logs ftw.

By running cat audit.log | grep "p.auster" | grep Secret | grep password we can see that passwords are stored in the Audit Logs, because they store the complete content of Secrets. It's never a good idea to reveal passwords in logs. In this case it would probably be sufficient to only store Metadata level information of Secrets which can be controlled via a Audit Policy.

Question 19 | Immutable Root FileSystem

Task weight: 2%

Use context: kubectl config use-context workload-prod

The Deployment immutable-deployment in Namespace team-purple should run immutable, it's created from file /opt/course/19/immutable-deployment.yaml. Even after a successful break-in, it shouldn't be possible for an attacker to modify the filesystem of the running container.

Modify the Deployment in a way that no processes inside the container can modify the local filesystem, only /tmp directory should be writeable. Don't modify the Docker image.

Save the updated YAML under /opt/course/19/immutable-deployment-new.yaml and update the running Deployment.

Answer:

Processes in containers can write to the local filesystem by default. This increases the attack surface when a non-malicious process gets hijacked. Preventing applications to write to disk or only allowing to certain directories can mitigate the risk. If there is for example a bug in Nginx which allows an attacker to override any file inside the container, then this only works if the Nginx process itself can write to the filesystem in the first place.

Making the root filesystem readonly can be done in the Docker image itself or in a Pod declaration.

Let us first check the Deployment immutable-deployment in Namespace team-purple:

➜ k -n team-purple edit deploy -o yaml

kubectl -n team-purple edit deploy -o yaml

apiVersion: apps/v1 kind: Deployment metadata: namespace: team-purple name: immutable-deployment labels: app: immutable-deployment ... spec: replicas: 1 selector: matchLabels: app: immutable-deployment template: metadata: labels: app: immutable-deployment spec: containers: - image: busybox:1.32.0 command: ['sh', '-c', 'tail -f /dev/null'] imagePullPolicy: IfNotPresent name: busybox restartPolicy: Always ... The container has write access to the Root File System, as there are no restrictions defined for the Pods or containers by an existing SecurityContext. And based on the task we're not allowed to alter the Docker image.

So we modify the YAML manifest to include the required changes:

cp /opt/course/19/immutable-deployment.yaml /opt/course/19/immutable-deployment-new.yaml

vim /opt/course/19/immutable-deployment-new.yaml

/opt/course/19/immutable-deployment-new.yaml

apiVersion: apps/v1 kind: Deployment metadata: namespace: team-purple name: immutable-deployment labels: app: immutable-deployment spec: replicas: 1 selector: matchLabels: app: immutable-deployment template: metadata: labels: app: immutable-deployment spec: containers: - image: busybox:1.32.0 command: ['sh', '-c', 'tail -f /dev/null'] imagePullPolicy: IfNotPresent name: busybox securityContext: # add readOnlyRootFilesystem: true # add volumeMounts: # add - mountPath: /tmp # add name: temp-vol # add volumes: # add - name: temp-vol # add emptyDir: {} # add restartPolicy: Always SecurityContexts can be set on Pod or container level, here the latter was asked. Enforcing readOnlyRootFilesystem: true will render the root filesystem readonly. We can then allow some directories to be writable by using an emptyDir volume.

Once the changes are made, let us update the Deployment:

➜ k delete -f /opt/course/19/immutable-deployment-new.yaml deployment.apps "immutable-deployment" deleted

➜ k create -f /opt/course/19/immutable-deployment-new.yaml deployment.apps/immutable-deployment created We can verify if the required changes are propagated:

➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- touch /abc.txt touch: /abc.txt: Read-only file system command terminated with exit code 1

➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- touch /var/abc.txt touch: /var/abc.txt: Read-only file system command terminated with exit code 1

➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- touch /etc/abc.txt touch: /etc/abc.txt: Read-only file system command terminated with exit code 1

➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- touch /tmp/abc.txt

➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- ls /tmp abc.txt The Deployment has been updated so that the container's file system is read-only, and the updated YAML has been placed under the required location. Sweet!

Question 20 | Update Kubernetes

Task weight: 8%

Use context: kubectl config use-context workload-stage

The cluster is running Kubernetes 1.27.6, update it to 1.28.2.

Use apt package manager and kubeadm for this.

Use ssh cluster3-controlplane1 and ssh cluster3-node1 to connect to the instances.

Answer:

Let's have a look at the current versions:

➜ k get node cluster3-controlplane1 Ready control-plane 96m v1.27.6 cluster3-node1 Ready 91m v1.27.6

Control Plane Master Components

First we should update the control plane components running on the master node, so we drain it:

➜ k drain cluster3-controlplane1 --ignore-daemonsets Next we ssh into it and check versions:

➜ ssh cluster3-controlplane1

➜ root@cluster3-controlplane1:~# kubelet --version Kubernetes v1.27.6

➜ root@cluster3-controlplane1:~# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2f", GitTreeState:"clean", BuildDate:"2023-09-13T09:34:32Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"} We see above that kubeadm is already installed in the required version. Else we would need to install it:

not necessary because here kubeadm is already installed in correct version

apt-mark unhold kubeadm apt-mark hold kubectl kubelet apt install kubeadm=1.28.2-00 apt-mark hold kubeadm Check what kubeadm has available as an upgrade plan:

➜ root@cluster3-controlplane1:~# kubeadm upgrade plan [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.27.6 [upgrade/versions] kubeadm version: v1.28.2 [upgrade/versions] Target version: v1.28.2 [upgrade/versions] Latest version in the v1.27 series: v1.27.6

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT TARGET kubelet 2 x v1.27.6 v1.28.2

Upgrade to the latest stable version:

COMPONENT CURRENT TARGET kube-apiserver v1.27.6 v1.28.2 kube-controller-manager v1.27.6 v1.28.2 kube-scheduler v1.27.6 v1.28.2 kube-proxy v1.27.6 v1.28.2 CoreDNS v1.10.1 v1.10.1 etcd 3.5.7-0 3.5.9-0

You can now apply the upgrade by executing the following command:

    kubeadm upgrade apply v1.28.2

The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no

And we apply to the required version:

➜ root@cluster3-controlplane1:~# kubeadm upgrade apply v1.28.2 [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade/version] You have chosen to change the cluster version to "v1.28.2" [upgrade/versions] Cluster version: v1.27.6 [upgrade/versions] kubeadm version: v1.28.2 [upgrade] Are you sure you want to proceed? [y/N]: y [upgrade/prepull] Pulling images required for setting up a Kubernetes cluster ... [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.28.2". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

Next we can check if our required version was installed correctly:

➜ root@cluster3-controlplane1:~# kubeadm upgrade plan [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.28.2 [upgrade/versions] kubeadm version: v1.28.2 [upgrade/versions] Target version: v1.28.2 [upgrade/versions] Latest version in the v1.28 series: v1.28.2

Control Plane kubelet and kubectl

Now we have to upgrade kubelet and kubectl:

➜ root@cluster3-controlplane1:~# apt update Hit:1 http://ppa.launchpad.net/rmescandon/yq/ubuntu focal InRelease Hit:3 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease Reading package lists... Done
Building dependency tree
Reading state information... Done 2 packages can be upgraded. Run 'apt list --upgradable' to see them.

➜ root@cluster3-controlplane1:~# apt-mark unhold kubelet kubectl kubelet was already not hold. kubectl was already not hold.

➜ root@cluster3-controlplane1:~# apt install kubelet=1.28.2-00 kubectl=1.28.2-00 Reading package lists... Done Building dependency tree
Reading state information... Done The following packages will be upgraded: kubectl kubelet 2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 29.8 MB of archives. After this operation, 5,194 kB of additional disk space will be used. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.28.2-00 [10.3 MB] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.28.2-00 [19.5 MB] Fetched 29.8 MB in 2s (17.3 MB/s)
(Reading database ... 112527 files and directories currently installed.) Preparing to unpack .../kubectl_1.28.2-00_amd64.deb ... Unpacking kubectl (1.28.2-00) over (1.27.6-00) ... Preparing to unpack .../kubelet_1.28.2-00_amd64.deb ... Unpacking kubelet (1.28.2-00) over (1.27.6-00) ... Setting up kubectl (1.28.2-00) ... Setting up kubelet (1.28.2-00) ...

➜ root@cluster3-controlplane1:~# apt-mark hold kubelet kubectl kubelet set on hold. kubectl set on hold.

➜ root@cluster3-controlplane1:~# service kubelet restart

➜ root@cluster3-controlplane1:~# service kubelet status ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Tue 2023-09-26 12:52:25 UTC; 3s ago Docs: https://kubernetes.io/docs/home/ Main PID: 34030 (kubelet) Tasks: 11 (limit: 1066) Memory: 34.4M CGroup: /system.slice/kubelet.service ...

➜ root@cluster3-controlplane1:~# kubectl get node NAME STATUS ROLES AGE VERSION cluster3-controlplane1 Ready,SchedulingDisabled control-plane 150m v1.28.2 cluster3-node1 Ready 143m v1.27.6 Done, and uncordon:

➜ k uncordon cluster3-controlplane1 node/cluster3-controlplane1 uncordoned

Data Plane

➜ k get node NAME STATUS ROLES AGE VERSION cluster3-controlplane1 Ready control-plane 150m v1.28.2 cluster3-node1 Ready 143m v1.27.6 Our data plane consist of one single worker node, so let's update it. First thing is we should drain it:

k drain cluster3-node1 --ignore-daemonsets Next we ssh into it and upgrade kubeadm to the wanted version, or check if already done:

➜ ssh cluster3-node1

➜ root@cluster3-node1:~# apt update Hit:1 http://ppa.launchpad.net/rmescandon/yq/ubuntu focal InRelease Hit:3 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B] Fetched 8,993 B in 1s (17.8 kB/s) Reading package lists... Done Building dependency tree
Reading state information... Done 3 packages can be upgraded. Run 'apt list --upgradable' to see them.

➜ root@cluster3-node1:~# apt-mark unhold kubeadm kubeadm was already not hold.

➜ root@cluster3-node1:~# apt-mark hold kubectl kubelet kubectl set on hold. kubelet set on hold.

➜ root@cluster3-node1:~# apt install kubeadm=1.28.2-00 Reading package lists... Done Building dependency tree
Reading state information... Done The following packages will be upgraded: kubeadm 1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. Need to get 10.3 MB of archives. After this operation, 2,589 kB of additional disk space will be used. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.28.2-00 [10.3 MB] Fetched 10.3 MB in 1s (19.1 MB/s) (Reading database ... 112527 files and directories currently installed.) Preparing to unpack .../kubeadm_1.28.2-00_amd64.deb ... Unpacking kubeadm (1.28.2-00) over (1.27.6-00) ... Setting up kubeadm (1.28.2-00) ...

➜ root@cluster3-node1:~# apt-mark hold kubeadm kubeadm set on hold.

➜ root@cluster3-node1:~# kubeadm upgrade node [upgrade] Reading configuration from the cluster... [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks [preflight] Skipping prepull. Not a control plane node. [upgrade] Skipping phase. Not a control plane node. [upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1123040998/config.yaml [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [upgrade] The configuration for this node was successfully updated! [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager. Now we follow that kubeadm told us in the last line and upgrade kubelet (and kubectl):

➜ root@cluster3-node1:~# apt-mark unhold kubectl kubelet Canceled hold on kubectl. Canceled hold on kubelet.

➜ root@cluster3-node1:~# apt install kubelet=1.28.2-00 kubectl=1.28.2-00 Reading package lists... Done Building dependency tree
Reading state information... Done The following packages will be upgraded: kubectl kubelet 2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 29.8 MB of archives. After this operation, 5,194 kB of additional disk space will be used. Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.28.2-00 [10.3 MB] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.28.2-00 [19.5 MB] Fetched 29.8 MB in 2s (14.5 MB/s)
(Reading database ... 112527 files and directories currently installed.) Preparing to unpack .../kubectl_1.28.2-00_amd64.deb ... Unpacking kubectl (1.28.2-00) over (1.27.6-00) ... Preparing to unpack .../kubelet_1.28.2-00_amd64.deb ... Unpacking kubelet (1.28.2-00) over (1.27.6-00) ... Setting up kubectl (1.28.2-00) ... Setting up kubelet (1.28.2-00) ...

➜ root@cluster3-node1:~# service kubelet restart

➜ root@cluster3-node1:~# service kubelet status ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Tue 2023-09-26 12:56:19 UTC; 4s ago Docs: https://kubernetes.io/docs/home/ Main PID: 34075 (kubelet) Tasks: 9 (limit: 1066) Memory: 26.4M CGroup: /system.slice/kubelet.service ... Looking good, what does the node status say?

➜ k get node NAME STATUS ROLES AGE VERSION cluster3-controlplane1 Ready control-plane 154m v1.28.2 cluster3-node1 Ready,SchedulingDisabled 147m v1.28.2 Beautiful, let's make it schedulable again:

➜ k uncordon cluster3-node1 node/cluster3-node1 uncordoned

➜ k get node NAME STATUS ROLES AGE VERSION cluster3-controlplane1 Ready control-plane 154m v1.28.2 cluster3-node1 Ready 147m v1.28.2 We're up to date.

Question 21 | Image Vulnerability Scanning

Task weight: 2%

(can be solved in any kubectl context)

The Vulnerability Scanner trivy is installed on your main terminal. Use it to scan the following images for known CVEs:

nginx:1.16.1-alpine k8s.gcr.io/kube-apiserver:v1.18.0 k8s.gcr.io/kube-controller-manager:v1.18.0 docker.io/weaveworks/weave-kube:2.7.0 Write all images that don't contain the vulnerabilities CVE-2020-10878 or CVE-2020-1967 into /opt/course/21/good-images.

Answer:

The tool trivy is very simple to use, it compares images against public databases.

➜ trivy nginx:1.16.1-alpine 2020-10-09T20:59:39.198Z INFO Need to update DB 2020-10-09T20:59:39.198Z INFO Downloading DB... 18.81 MiB / 18.81 MiB [------------------------------------- 2020-10-09T20:59:45.499Z INFO Detecting Alpine vulnerabilities...

nginx:1.16.1-alpine (alpine 3.10.4)

Total: 7 (UNKNOWN: 0, LOW: 0, MEDIUM: 7, HIGH: 0, CRITICAL: 0)

➜ trivy docker.io/weaveworks/weave-kube:2.7.0 | grep -E 'CVE-2020-10878|CVE-2020-1967' ➜ The only image without the any of the two CVEs is docker.io/weaveworks/weave-kube:2.7.0, hence our answer will be:

/opt/course/21/good-images

docker.io/weaveworks/weave-kube:2.7.0

Question 22 | Manual Static Security Analysis

Task weight: 3%

(can be solved in any kubectl context)

The Release Engineering Team has shared some YAML manifests and Dockerfiles with you to review. The files are located under /opt/course/22/files.

As a container security expert, you are asked to perform a manual static analysis and find out possible security issues with respect to unwanted credential exposure. Running processes as root is of no concern in this task.

Write the filenames which have issues into /opt/course/22/security-issues.

NOTE: In the Dockerfile and YAML manifests, assume that the referred files, folders, secrets and volume mounts are present. Disregard syntax or logic errors.

Answer:

We check location /opt/course/22/files and list the files.

➜ ls -la /opt/course/22/files total 48 drwxr-xr-x 2 k8s k8s 4096 Sep 16 19:08 . drwxr-xr-x 3 k8s k8s 4096 Sep 16 19:08 .. -rw-r--r-- 1 k8s k8s 692 Sep 16 19:08 Dockerfile-go -rw-r--r-- 1 k8s k8s 897 Sep 16 19:08 Dockerfile-mysql -rw-r--r-- 1 k8s k8s 743 Sep 16 19:08 Dockerfile-py -rw-r--r-- 1 k8s k8s 341 Sep 16 19:08 deployment-nginx.yaml -rw-r--r-- 1 k8s k8s 705 Sep 16 19:08 deployment-redis.yaml -rw-r--r-- 1 k8s k8s 392 Sep 16 19:08 pod-nginx.yaml -rw-r--r-- 1 k8s k8s 228 Sep 16 19:08 pv-manual.yaml -rw-r--r-- 1 k8s k8s 188 Sep 16 19:08 pvc-manual.yaml -rw-r--r-- 1 k8s k8s 211 Sep 16 19:08 sc-local.yaml -rw-r--r-- 1 k8s k8s 902 Sep 16 19:08 statefulset-nginx.yaml We have 3 Dockerfiles and 7 Kubernetes Resource YAML manifests. Next we should go over each to find security issues with the way credentials have been used.

NOTE: You should be comfortable with Docker Best Practices and the Kubernetes Configuration Best Practices.

While navigating through the files we might notice:

Number 1

File Dockerfile-mysql might look innocent on first look. It copies a file secret-token over, uses it and deletes it afterwards. But because of the way Docker works, every RUN, COPY and ADD command creates a new layer and every layer is persistet in the image.

This means even if the file secret-token get's deleted in layer Z, it's still included with the image in layer X and Y. In this case it would be better to use for example variables passed to Docker.

/opt/course/22/files/Dockerfile-mysql

FROM ubuntu

Add MySQL configuration

COPY my.cnf /etc/mysql/conf.d/my.cnf COPY mysqld_charset.cnf /etc/mysql/conf.d/mysqld_charset.cnf

RUN apt-get update &&
apt-get -yq install mysql-server-5.6 &&

Add MySQL scripts

COPY import_sql.sh /import_sql.sh COPY run.sh /run.sh

Configure credentials

COPY secret-token . # LAYER X RUN /etc/register.sh ./secret-token # LAYER Y RUN rm ./secret-token # delete secret token again # LATER Z

EXPOSE 3306 CMD ["/run.sh"] So we do:

echo Dockerfile-mysql >> /opt/course/22/security-issues

Number 2

The file deployment-redis.yaml is fetching credentials from a Secret named mysecret and writes these into environment variables. So far so good, but in the command of the container it's echoing these which can be directly read by any user having access to the logs.

/opt/course/22/files/deployment-redis.yaml

apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: mycontainer image: redis command: ["/bin/sh"] args: - "-c" - "echo $SECRET_USERNAME && echo $SECRET_PASSWORD && docker-entrypoint.sh" # NOT GOOD env: - name: SECRET_USERNAME valueFrom: secretKeyRef: name: mysecret key: username - name: SECRET_PASSWORD valueFrom: secretKeyRef: name: mysecret key: password Credentials in logs is never a good idea, hence we do:

echo deployment-redis.yaml >> /opt/course/22/security-issues

Number 3

In file statefulset-nginx.yaml, the password is directly exposed in the environment variable definition of the container.

/opt/course/22/files/statefulset-nginx.yaml

... apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: serviceName: "nginx" replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: k8s.gcr.io/nginx-slim:0.8 env: - name: Username value: Administrator - name: Password value: MyDiReCtP@sSw0rd # NOT GOOD ports: - containerPort: 80 name: web .. This should better be injected via a Secret. So we do:

echo statefulset-nginx.yaml >> /opt/course/22/security-issues

➜ cat /opt/course/22/security-issues Dockerfile-mysql deployment-redis.yaml statefulset-nginx.yaml

CKS Simulator Preview Kubernetes 1.28 https://killer.sh

This is a preview of the full CKS Simulator course content.

The full course contains 22 questions and scenarios which cover all the CKS areas. The course also provides a browser terminal which is a very close replica of the original one. This is great to get used and comfortable before the real exam. After the test session (120 minutes), or if you stop it early, you'll get access to all questions and their detailed solutions. You'll have 36 hours cluster access in total which means even after the session, once you have the solutions, you can still play around.

The following preview will give you an idea of what the full course will provide. These preview questions are not part of the 22 in the full course but in addition to it. But the preview questions are part of the same CKS simulation environment which we setup for you, so with access to the full course you can solve these too.

The answers provided here assume that you did run the initial terminal setup suggestions as provided in the tips section, but especially:

alias k=kubectl

export do="-o yaml --dry-run=client"

These questions can be solved in the test environment provided through the CKS Simulator

Preview Question 1

Use context: kubectl config use-context infra-prod

You have admin access to cluster2. There is also context gianna@infra-prod which authenticates as user gianna with the same cluster.

There are existing cluster-level RBAC resources in place to, among other things, ensure that user gianna can never read Secret contents cluster-wide. Confirm this is correct or restrict the existing RBAC resources to ensure this.

I addition, create more RBAC resources to allow user gianna to create Pods and Deployments in Namespaces security, restricted and internal. It's likely the user will receive these exact permissions as well for other Namespaces in the future.

Answer:

Part 1 - check existing RBAC rules

We should probably first have a look at the existing RBAC resources for user gianna. We don't know the resource names but we know these are cluster-level so we can search for a ClusterRoleBinding:

k get clusterrolebinding -oyaml | grep gianna -A10 -B20 From this we see the binding is also called gianna:

k edit clusterrolebinding gianna

kubectl edit clusterrolebinding gianna

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: "2020-09-26T13:57:58Z" name: gianna resourceVersion: "3049" selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/gianna uid: 72b64a3b-5958-4cf8-8078-e5be2c55b25d roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: gianna subjects:

apiGroup: rbac.authorization.k8s.io kind: User name: gianna It links user gianna to same named ClusterRole:

k edit clusterrole gianna

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: "2020-09-26T13:57:55Z" name: gianna resourceVersion: "3038" selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/gianna uid: b713c1cf-87e5-4313-808e-1a51f392adc0 rules:

apiGroups:
- "" resources:
- secrets
- configmaps
- pods
- namespaces verbs:
- list According to the task the user should never be able to read Secrets content. They verb list might indicate on first look that this is correct. We can also check using K8s User Impersonation:

➜ k auth can-i list secrets --as gianna yes

➜ k auth can-i get secrets --as gianna no But let's have a closer look:

➜ k config use-context gianna@infra-prod Switched to context "gianna@infra-prod".

➜ k -n security get secrets NAME TYPE DATA AGE default-token-gn455 kubernetes.io/service-account-token 3 20m kubeadmin-token Opaque 1 20m mysql-admin Opaque 1 20m postgres001 Opaque 1 20m postgres002 Opaque 1 20m vault-token Opaque 1 20m

➜ k -n security get secret kubeadmin-token Error from server (Forbidden): secrets "kubeadmin-token" is forbidden: User "gianna" cannot get resource "secrets" in API group "" in the namespace "security" Still all expected, but being able to list resources also allows to specify the format:

➜ k -n security get secrets -oyaml | grep password password: ekhHYW5lQUVTaVVxCg== {"apiVersion":"v1","data":{"password":"ekhHYW5lQUVTaVVxCg=="},"kind":"Secret","metadata":{"annotations":{},"name":"kubeadmin-token","namespace":"security"},"type":"Opaque"} f:password: {} password: bWdFVlBSdEpEWHBFCg== ... The user gianna is actually able to read Secret content. To prevent this we should remove the ability to list these:

k config use-context infra-prod # back to admin context

k edit clusterrole gianna

kubectl edit clusterrole gianna

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: "2020-09-26T13:57:55Z" name: gianna resourceVersion: "4496" selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/gianna uid: b713c1cf-87e5-4313-808e-1a51f392adc0 rules:

apiGroups:
- "" resources:

- secrets # remove

configmaps
pods
namespaces verbs:
list

Part 2 - create additional RBAC rules

Let's talk a little about RBAC resources:

A ClusterRole|Role defines a set of permissions and where it is available, in the whole cluster or just a single Namespace.

A ClusterRoleBinding|RoleBinding connects a set of permissions with an account and defines where it is applied, in the whole cluster or just a single Namespace.

Because of this there are 4 different RBAC combinations and 3 valid ones:

Role + RoleBinding (available in single Namespace, applied in single Namespace) ClusterRole + ClusterRoleBinding (available cluster-wide, applied cluster-wide) ClusterRole + RoleBinding (available cluster-wide, applied in single Namespace) Role + ClusterRoleBinding (NOT POSSIBLE: available in single Namespace, applied cluster-wide)

The user gianna should be able to create Pods and Deployments in three Namespaces. We can use number 1 or 3 from the list above. But because the task says: "The user might receive these exact permissions as well for other Namespaces in the future", we choose number 3 as it requires to only create one ClusterRole instead of three Roles.

k create clusterrole gianna-additional --verb=create --resource=pods --resource=deployments This will create a ClusterRole like:

kubectl create clusterrole gianna-additional --verb=create --resource=pods --resource=deployments

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: null name: gianna-additional rules:

apiGroups:
- "" resources:
- pods verbs:
- create
apiGroups:
- apps resources:
- deployments verbs:
- create Next the three bindings:

k -n security create rolebinding gianna-additional
--clusterrole=gianna-additional --user=gianna

k -n restricted create rolebinding gianna-additional
--clusterrole=gianna-additional --user=gianna

k -n internal create rolebinding gianna-additional
--clusterrole=gianna-additional --user=gianna Which will create RoleBindings like:

k -n security create rolebinding gianna-additional --clusterrole=gianna-additional --user=gianna

apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: gianna-additional namespace: security roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: gianna-additional subjects:

apiGroup: rbac.authorization.k8s.io kind: User name: gianna And we test:

➜ k -n default auth can-i create pods --as gianna no

➜ k -n security auth can-i create pods --as gianna yes

➜ k -n restricted auth can-i create pods --as gianna yes

➜ k -n internal auth can-i create pods --as gianna yes Feel free to verify this as well by actually creating Pods and Deployments as user gianna through context gianna@infra-prod.

Preview Question 2

Use context: kubectl config use-context infra-prod

There is an existing Open Policy Agent + Gatekeeper policy to enforce that all Namespaces need to have label security-level set. Extend the policy constraint and template so that all Namespaces also need to set label management-team. Any new Namespace creation without these two labels should be prevented.

Write the names of all existing Namespaces which violate the updated policy into /opt/course/p2/fix-namespaces.

Answer:

We look at existing OPA constraints, these are implemeted using CRDs by Gatekeeper:

➜ k get crd NAME CREATED AT blacklistimages.constraints.gatekeeper.sh 2020-09-14T19:29:31Z configs.config.gatekeeper.sh 2020-09-14T19:29:04Z constraintpodstatuses.status.gatekeeper.sh 2020-09-14T19:29:05Z constrainttemplatepodstatuses.status.gatekeeper.sh 2020-09-14T19:29:05Z constrainttemplates.templates.gatekeeper.sh 2020-09-14T19:29:05Z requiredlabels.constraints.gatekeeper.sh 2020-09-14T19:29:31Z So we can do:

➜ k get constraint NAME AGE blacklistimages.constraints.gatekeeper.sh/pod-trusted-images 10m

NAME AGE requiredlabels.constraints.gatekeeper.sh/namespace-mandatory-labels 10m And check violations for the namespace-mandatory-label one, which we can do in the resource status:

➜ k describe requiredlabels namespace-mandatory-labels Name: namespace-mandatory-labels Namespace:
Labels: Annotations: API Version: constraints.gatekeeper.sh/v1beta1 Kind: RequiredLabels ... Status: ... Total Violations: 1 Violations: Enforcement Action: deny Kind: Namespace Message: you must provide labels: {"security-level"} Name: sidecar-injector Events:
We see one violation for Namespace "sidecar-injector". Let's get an overview over all Namespaces:

➜ k get ns --show-labels NAME STATUS AGE LABELS default Active 21m management-team=green,security-level=high gatekeeper-system Active 14m admission.gatekeeper.sh/ignore=no-self-managing,control-plane=controller-manager,gatekeeper.sh/system=yes,management-team=green,security-level=high jeffs-playground Active 14m security-level=high kube-node-lease Active 21m management-team=green,security-level=high kube-public Active 21m management-team=red,security-level=low kube-system Active 21m management-team=green,security-level=high restricted Active 14m management-team=blue,security-level=medium security Active 14m management-team=blue,security-level=medium sidecar-injector Active 14m When we try to create a Namespace without the required label we get an OPA error:

➜ k create ns test Error from server ([denied by namespace-mandatory-labels] you must provide labels: {"security-level"}): error when creating "ns.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by namespace-mandatory-labels] you must provide labels: {"security-level"} Next we edit the constraint to add another required label:

k edit requiredlabels namespace-mandatory-labels

kubectl edit requiredlabels namespace-mandatory-labels

apiVersion: constraints.gatekeeper.sh/v1beta1 kind: RequiredLabels metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"RequiredLabels","metadata":{"annotations":{},"name":"namespace-mandatory-labels"},"spec":{"match":{"kinds":[{"apiGroups":[""],"kinds":["Namespace"]}]},"parameters":{"labels":["security-level"]}}} creationTimestamp: "2020-09-14T19:29:53Z" generation: 1 name: namespace-mandatory-labels resourceVersion: "3081" selfLink: /apis/constraints.gatekeeper.sh/v1beta1/requiredlabels/namespace-mandatory-labels uid: 2a51a291-e07f-4bab-b33c-9b8c90e5125b spec: match: kinds: - apiGroups: - "" kinds: - Namespace parameters: labels: - security-level - management-team # add As we can see the constraint is using kind: RequiredLabels as template, which is a CRD created by Gatekeeper. Let's apply the change and see what happens (give OPA a minute to apply the changes internally):

➜ k describe requiredlabels namespace-mandatory-labels ... Violations: Enforcement Action: deny Kind: Namespace Message: you must provide labels: {"management-team"} Name: jeffs-playground After the changes we can see that now another Namespace jeffs-playground is in trouble. Because that one only specifies one required label. But what about the earlier violation of Namespace sidecar-injector?

➜ k get ns --show-labels NAME STATUS AGE LABELS default Active 21m management-team=green,security-level=high gatekeeper-system Active 17m admission.gatekeeper.sh/ignore=no-self-managing,control-plane=controller-manager,gatekeeper.sh/system=yes,management-team=green,security-level=high jeffs-playground Active 17m security-level=high kube-node-lease Active 21m management-team=green,security-level=high kube-public Active 21m management-team=red,security-level=low kube-system Active 21m management-team=green,security-level=high restricted Active 17m management-team=blue,security-level=medium security Active 17m management-team=blue,security-level=medium sidecar-injector Active 17m Namespace sidecar-injector should also be in trouble, but it isn't any longer. This doesn't seem right, it means we could still create Namespaces without any labels just like using k create ns test.

So we check the template:

➜ k get constrainttemplates NAME AGE blacklistimages 20m requiredlabels 20m

➜ k edit constrainttemplates requiredlabels

kubectl edit constrainttemplates requiredlabels

apiVersion: templates.gatekeeper.sh/v1beta1 kind: ConstraintTemplate ... spec: crd: spec: names: kind: RequiredLabels validation: openAPIV3Schema: properties: labels: items: string type: array targets:

rego: | package k8srequiredlabels

violation[{"msg": msg, "details": {"missing_labels": missing}}] { provided := {label | input.review.object.metadata.labels[label]} required := {label | label := input.parameters.labels[_]} missing := required - provided # count(missing) == 1 # WRONG count(missing) > 0 msg := sprintf("you must provide labels: %v", [missing]) } target: admission.k8s.gatekeeper.sh In the rego script we need to change count(missing) == 1 to count(missing) > 0 . If we don't do this then the policy only complains if there is one missing label, but there can be multiple missing ones.

After waiting a bit we check the constraint again:

➜ k describe requiredlabels namespace-mandatory-labels ... Total Violations: 2 Violations: Enforcement Action: deny Kind: Namespace Message: you must provide labels: {"management-team"} Name: jeffs-playground Enforcement Action: deny Kind: Namespace Message: you must provide labels: {"security-level", "management-team"} Name: sidecar-injector Events: This looks better. Finally we write the Namespace names with violations into the required location:

/opt/course/p2/fix-namespaces

sidecar-injector jeffs-playground

Preview Question 3

Use context: kubectl config use-context workload-stage

A security scan result shows that there is an unknown miner process running on one of the Nodes in cluster3. The report states that the process is listening on port 6666. Kill the process and delete the binary.

Answer:

We have a look at existing Nodes:

➜ k get node NAME STATUS ROLES AGE VERSION cluster3-controlplane1 Ready control-plane 109m v1.27.1 cluster3-node1 Ready 105m v1.27.1 First we check the master:

➜ ssh cluster3-controlplane1

➜ root@cluster3-controlplane1:~# netstat -plnt | grep 6666

➜ root@cluster3-controlplane1:~# Doesn't look like any process listening on this port. So we check the worker:

➜ ssh cluster3-node1

➜ root@cluster3-node1:~# netstat -plnt | grep 6666 tcp6 0 0 :::6666 :::* LISTEN 9591/system-atm
There we go! We could also use lsof:

➜ root@cluster3-node1:~# lsof -i :6666 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME system-at 9591 root 3u IPv6 47760 0t0 TCP *:6666 (LISTEN) Before we kill the process we can check the magic /proc directory for the full process path:

➜ root@cluster3-node1:~# ls -lh /proc/9591/exe lrwxrwxrwx 1 root root 0 Sep 26 16:10 /proc/9591/exe -> /bin/system-atm So we finish it:

➜ root@cluster3-node1:~# kill -9 9591

➜ root@cluster3-node1:~# rm /bin/system-atm Done.

CKS Tips Kubernetes 1.28 In this section we'll provide some tips on how to handle the CKS exam and browser terminal.

Knowledge

Pre-Knowledge

You should have your CKA knowledge up to date and be fast with kubectl, so we suggest to do:

Study all scenarios on https://killercoda.com/killer-shell-cka Knowledge

Study all topics as proposed in the curriculum till you feel comfortable with all. Study all scenarios on https://killercoda.com/killer-shell-cks Read the free Sysdig Kubernetes Security Guide Also a nice read (though based on outdated k8s version) is the Kubernetes Security book by Liz Rice Check out the Cloud Native Security Whitepaper Great repository with many tips and sources: walidshari Approach

Do 1 or 2 test session with this CKS Simulator. Understand the solutions and maybe try out other ways to achieve the same thing. Setup your aliases, be fast and breath kubectl Content

Be comfortable with changing the kube-apiserver in a kubeadm setup Be able to work with AdmissionControllers Know how to create and use the ImagePolicyWebhook Know how to use opensource tools Falco, Sysdig, Tracee, Trivy

CKS Exam Info

Read the Curriculum

https://github.com/cncf/curriculum

Read the Handbook

https://docs.linuxfoundation.org/tc-docs/certification/lf-handbook2

Read the important tips

https://docs.linuxfoundation.org/tc-docs/certification/important-instructions-cks

Read the FAQ

https://docs.linuxfoundation.org/tc-docs/certification/faq-cka-ckad-cks

Kubernetes documentation

Get familiar with the Kubernetes documentation and be able to use the search. Allowed links are:

https://kubernetes.io/docs https://github.com/kubernetes https://kubernetes.io/blog https://aquasecurity.github.io/trivy https://falco.org/docs https://gitlab.com/apparmor/apparmor/-/wikis/Documentation NOTE: Verify the list here

CKS clusters

In the CKS exam you'll get access to as many clusters as you have questions, each will be solved in its own cluster. This is great because you cannot interfere with other tasks by breaking one. Every cluster will have one master and one worker node.

The Test Environment / Browser Terminal

You'll be provided with a browser terminal which uses Ubuntu 20. The standard shells included with a minimal install of Ubuntu 20 will be available, including bash.

Laggin

There could be some lagging, definitely make sure you are using a good internet connection because your webcam and screen are uploading all the time.

Kubectl autocompletion and commands

Autocompletion is configured by default, as well as the k alias source and others:

kubectl with k alias and Bash autocompletion

yq and jqfor YAML/JSON processing

tmux for terminal multiplexing

curl and wget for testing web services

man and man pages for further documentation

Copy & Paste

There could be issues copying text (like pod names) from the left task information into the terminal. Some suggested to "hard" hit or long hold Cmd/Ctrl+C a few times to take action. Apart from that copy and paste should just work like in normal terminals.

Percentages and Score

There are 15-20 questions in the exam and 100% of total percentage to reach. Each questions shows the % it gives if you solve it. Your results will be automatically checked according to the handbook. If you don't agree with the results you can request a review by contacting the Linux Foundation support.

Notepad & Skipping Questions

You have access to a simple notepad in the browser which can be used for storing any kind of plain text. It makes sense to use this for saving skipped question numbers and their percentages. This way it's possible to move some questions to the end. It might make sense to skip 2% or 3% questions and go directly to higher ones.

Contexts

You'll receive access to various different clusters and resources in each. They provide you the exact command you need to run to connect to another cluster/context. But you should be comfortable working in different namespaces with kubectl.

PSI Bridge

Starting with PSI Bridge:

The exam will now be taken using the PSI Secure Browser, which can be downloaded using the newest versions of Microsoft Edge, Safari, Chrome, or Firefox Multiple monitors will no longer be permitted Use of personal bookmarks will no longer be permitted The new ExamUI includes improved features such as:

A remote desktop configured with the tools and software needed to complete the tasks A timer that displays the actual time remaining (in minutes) and provides an alert with 30, 15, or 5 minute remaining The content panel remains the same (presented on the Left Hand Side of the ExamUI) Read more here.

Browser Terminal Setup

It should be considered to spend ~1 minute in the beginning to setup your terminal. In the real exam the vast majority of questions will be done from the main terminal. For few you might need to ssh into another machine. Just be aware that configurations to your shell will not be transferred in this case.

Minimal Setup

Alias

The alias k for kubectl will already be configured together with autocompletion. In case not you can configure it using this link.

Vim

The following settings will already be configured in your real exam environment in ~/.vimrc. But it can never hurt to be able to type these down:

set tabstop=2 set expandtab set shiftwidth=2 The expandtab make sure to use spaces for tabs. Memorize these and just type them down. You can't have any written notes with commands on your desktop etc.

Optional Setup

Fast dry-run output

export do="--dry-run=client -o yaml" This way you can just run k run pod1 --image=nginx $do. Short for "dry output", but use whatever name you like.

Fast pod delete

export now="--force --grace-period 0" This way you can run k delete pod1 $now and don't have to wait for ~30 seconds termination time.

Persist bash settings

You can store aliases and other setup in ~/.bashrc if you're planning on using different shells or tmux.

Alias Namespace

In addition you could define an alias like:

alias kn='kubectl config set-context --current --namespace ' Which allows you to define the default namespace of the current context. Then once you switch a context or namespace you can just run:

kn default # set default to default kn my-namespace # set default to my-namespace But only do this if you used it before and are comfortable doing so. Else you need to specify the namespace for every call, which is also fine:

k -n my-namespace get all k -n my-namespace get pod ...

Be fast

Use the history command to reuse already entered commands or use even faster history search through Ctrl r .

If a command takes some time to execute, like sometimes kubectl delete pod x. You can put a task in the background using Ctrl z and pull it back into foreground running command fg.

You can delete pods fast with:

k delete pod x --grace-period 0 --force

k delete pod x $now # if export from above is configured

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
README.md		README.md

hstiwana/cks

Folders and files

Latest commit

History

Repository files navigation

/opt/course/1/contexts

/opt/course/1/cert

/etc/falco/falco.yaml

Where security notifications should go.

Multiple outputs can be enabled.

Container is supposed to be immutable. Package management should be done in building the image.

Container is supposed to be immutable. Package management should be done in building the image.

/opt/course/2/falco.log

/etc/kubernetes/manifests/kube-apiserver.yaml

- --kubernetes-service-node-port=31000 # delete or set to 0

kubectl edit namespace team-red

/opt/course/4/logs

/etc/kubernetes/manifests/kube-controller-manager.yaml

/var/lib/kubelet/config.yaml

./compare

kubectl edit blacklistimages pod-trusted-images

kubectl edit constrainttemplates blacklistimages

kubectl -n team-blue edit deploy image-verify

/etc/kubernetes/manifests/kube-apiserver.yaml

/etc/kubernetes/audit/policy.yaml

/etc/kubernetes/audit/policy.yaml

log Secret resources audits, level Metadata

log node related audits, level RequestResponse

for everything else don't log anything

shows Secret entries

confirms Secret entries are only of level Metadata

shows RequestResponse level entries

shows RequestResponse level entries are only for system:nodes

kubectl -n team-purple edit deploy -o yaml

/opt/course/19/immutable-deployment-new.yaml

not necessary because here kubeadm is already installed in correct version

nginx:1.16.1-alpine (alpine 3.10.4)

/opt/course/21/good-images

/opt/course/22/files/Dockerfile-mysql

Add MySQL configuration

Add MySQL scripts

Configure credentials

/opt/course/22/files/deployment-redis.yaml

/opt/course/22/files/statefulset-nginx.yaml

kubectl edit clusterrolebinding gianna

kubectl edit clusterrole gianna

- secrets # remove

kubectl create clusterrole gianna-additional --verb=create --resource=pods --resource=deployments

k -n security create rolebinding gianna-additional --clusterrole=gianna-additional --user=gianna

kubectl edit requiredlabels namespace-mandatory-labels

kubectl edit constrainttemplates requiredlabels

/opt/course/p2/fix-namespaces

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Packages